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Late Binding o£ Variables During Test Case 

Gen ration for Hardware and Software Design 

Verification 



5 BACKGROUND OF THE INVENTION 

1. Field of the Invention, 

This invention generally relates to design verifica- 
tion, and more specifically relates to test program gen- 
eration in a context of a computer mechanism and methods 
10 for testing a design for compliance with the design 
specification. 

2. Description of the Related Art. 

An important aspect of designing a hardware or 
software design, for example an advanced computer 

15 processor, is the ability to test the design of the 
processor thoroughly, in order to assure that the design 
complies with desired architectural, performance and 
design specifications. In current industrial practice, 
most of the verification is done by generating a very 

20 large number of tests by random test generators. 

Test program generators are basically sophisticated 
software engines, which are used to create numerous test 
cases. By appropriate configuration, it is possible for 
test generation to be focused on very specific ranges of 

25 conditions, or broadened to cover a wide range of logic. 
Today, large numbers of test cases can be created 
automatically in the time that a single test case could 
be written manually, as was done prior to the advent of 
test case generators. Modern test program generators are 

30 sophisticated enough to evaluate the microarchitectural 
implementation of a processor. 
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Typically, the input to the test program generator is 
a user-defined sequence of partially specified 
instructions, known as an instruction stream, which acts 
as a template for the test to be generated. The 
5 instruction stream is generally incomplete, in that 
various details of each instruction, such as the specific 
source and the target resources to be used, the data 
values of each uninitialized resource, and even the exact 
instruction to be generated, may be left unspecified. The 

10 test program generator then generates a complete test by 
filling in the missing information with random values. 
The choice of random values is often biased, so as to 
increase the likelihood of detecting a design flaw. The 
use of templates in this manner allows the creation of 

15 numerous test cases that stress the implementation of the 
logic, creating various conditions such as "buffer full", 
or "pipe stall" . 

An example of a conventional test program generator 
is the IBM tool, Genesys, which is disclosed in the docu- 

20 ment Model -Based Test Generation for Process Design Veri- 
fication, y. Lichtenstein at. al.. Sixth Innovative Ap- 
plications of Artificial Intelligence Conference, Au- 
gust 1994, pp. 83-94. An updated version, of Genesys, 
known as Genesys -Pro, is a generic random test generator, 

25 targeted at the architectural level and applicable to any 
architecture - 

Another conventional test program generator, AVPGEN, 
is disclosed in the document AVPGEN - A Generator for Ar- 
chitecture Verification Test Cases, A. Chandra, et al., 
30 IEEE Trans. Very Large Scale Integration (VLSI) Syst. 3, 
No. 2, pp. 188-200 (June 1995). 

Still another test generator is disclosed in the pub- 
lication B. O'Krafka, et al., MPTG: A Portable Test Gen- 
erator for Cache -Coherent Multiprocessors, International 
35 Conference on Computers and Communications, 1995, 
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pp. 38-44. This generator is capable of generating tests 
that include scenarios with unpredictable results and re- 
lies on checks carried out for intermediate points in the 
test. 

5 The principles of the present invention may be ap- 

plied to any of the above-noted test generators, as well 
as other test pattern generation tools, such as the veri- 
fication automation tool SpecMan™ available from 
VerisityTM, 2041 Landings Drive, Mountain View, CA 94043 

10 and the Vera Testbench Automation product, available from 
Synopsys, Inc. 700 East Middlefield Road, Mountain View, 
CA 94043. A method previously used to increase the capa- 
bility of test generators examines the state of the de- 
sign before generating the next input. In this method, 

15 often referred to as dynamic generation, the test genera- 
tor uses the state of the design to determine inputs hav- 
ing the best chances of producing an interesting test 
case. For example, if the goal of the test generator is 
to create resource interdependency in the pipelines of a 

20 processor, it generates instructions that use the same 
resources as the instructions already in the pipelines. 

An alternative approach to improve coverage uses sym- 
bolic simulation and symbolic trajectory evaluation tech- 
niques. These techniques are described, respectively, in 

25 the documents Improving Coverage Analysis and Test Gen- 
eration for Large Designs, J. Bergmann and M. Horowitz, 
Proceedings of the International Conference on Computer 
Design, pp. 580-583, November 1999, and Formal verifica- 
tion by Symbolic Evaluation of Partially-ordered Trajec- 

30 tories, C. J. H. Seger and R. E. Bryant, Formal Methods 
in System Design: An International Journal, 6(2): 147- 
189, March 1995. In symbolic simulation, some of the in- 
puts to the design are represented as symbols. These sym- 
bols propagate through the circuit to the outputs. The 

35 main advantage of symbolic simulation is that large 
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spaces can be covered in parallel, with a single symbolic 
input. However, this technique still suffers from explo- 
sion of the symbol representation, and thus it is limited 
to small designs. In addition, it is difficult for sym- 
5 bolic simulators to simulate complex functions and opera- 
tors, such as multiplication. 

A third approach builds test generators that include 
a precise model of the verified designs. Using this 
model, the test generator represents a desired event as a 

10 state-machine traversal problem or a constraint satisfac- 
tion problem. A solution of this problem yields a test 
case in which the desired event occurs. The main disad- 
vantage of this approach is that it requires considerable 
effort to specify of the model and build the generation 

15 engine. 

The use of such advanced random test generators may 
increase the quality of generated tests, but cannot 
ensure that all the ''dark corners" in the design are 
exposed, and that the entire design is adequately 

20 exercised. The term dark corners as used herein with 
respect to the design refers to events that are difficult 
to anticipate in practice, and for which test cases are 
difficult to construct using test generators. Typical 
dark corners include transient interdependencies of 

25 resources of a processor. Coverage analysis helps detect 
non-covered events, but does not help cover these events. 
Generation of interesting test cases that reach these 
dif f icult-to-cover dark corners is one of the main 
challenges in functional design verification. 

30 In the case of software designs, the main problem in 

testing concurrent programs is their nondeterminism: two 
executions of such a program may yield different results. 
Most of the work in the field of concurrent testing has 
been focused on detecting race conditions. However, race 

35 conditions have a low probability of manifesting 



IL920030044US1 



themselves, and even when they do it is not always an 
indication of a fault. 

One approach to testing software designs is disclosed 
in the documents 0. Edelstein, E. Farchi, Y. Nir, G. Rat- 
5 saby, and S. Ur., Multithreaded Java Program Test Genera- 
tion. IBM Systems Journal, 41 (1) : 111-125, 2002, and S. D. 
Stoller. Model-checking Multi-threaded Distributed Java 
Programs, in Proceedings of the 7th International SPIN 
Workshop on Model Checking of Software, pages 224-244, 

10 New York, 2000. Springer Verlag. The problem of generat- 
ing different interleaving for the purpose of revealing 
concurrent faults was approached by seeding the program 
with conditional sleep statements at shared memory access 
and synchronization events. At run time, random, biased 

15 random, or coverage-based decisions were taken as to 
whether to execute seeded primitives. 

SUmSKRY OF THE INVENTION 

One problem with the approach of dynamic generation 
is that in many cases, by the time the generated inputs 

20 progress to the area of interest, the state of the design 
has changed, and the reason that led to the generation of 
the inputs no longer exists. For instance, it is possible 
that the test generator, in an attempt to create interde- 
pendency in execution pipelines of a processor, assigns a 

25 register Rl, for example, because this register is cur- 
rently in use in the execution pipeline. However, by the 
time the generated instruction reaches the execution 
pipeline, the instruction that used the register Rl may 
have finished its execution. The requested interdepen- 

30 dency event never occurs. 

According to a disclosed embodiment of the inven- 
tion, late binding of variables is employed in a test 
generator in order to improve test coverage significantly 
with a reasonable performance penalty. 
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A test generator delays binding of values to vari- 
ables during generation of test cases until these values 
are needed by the design. Each variable in the design is 
associated with a Boolean flag that indicates if the re- 
5 source has a valid value, or has been evaluated such that 
its information content is known, or whether its informa- 
tion content is as yet undefined. Each such variable is 
associated with a linked list that points to other re- 
sources that should have the same value. Algorithms are 

10 presented, which enable the test generator to handle ac- 
cess to variables with the unidentified values, and de- 
ferring binding of the value until it becomes necessary. 
The technique is powerful enough to handle any arbitrary 
Boolean function. Experimentally/ the late binding tech- 

15 nique according to the invention improves the quality and 
rapidity of test coverage, with a reasonable performance 
penalty during simulation. 

The invention provides a method of verifying a de- 
sign, which is carried out by generating a test case for 

20 execution using the design. One of the resources in the 
design is required at a predetermined time to accommodate 
a signal during execution of the test case. The method is 
further carried out by designating the one resource as an 
unidentified resource, delaying binding the signal to the 

25 unidentified resource until immediately prior to the pre- 
determined time, thereafter binding the signal to the 
unidentified resource to define a bound resource, and ac- 
cessing the bound resource. 

In another aspect of the method, prior to the prede- 

30 termined time, the unidentified resource is copied to an- 
other resource, and after passage of the predetermined 
time, the signal and the other resource are bound to- 
gether . 
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According to an additional aspect of the method, the 
signal is an outcome determinative input to a Boolean 
function. 

One aspect of the method includes designating a sec- 
5 ond unidentified resource for accommodation of a second 
signal that is a second input of the Boolean function, 
wherein an output of the Boolean function is insensitive 
to the second input, and avoiding binding the second uni- 
dentified resource with the second signal during execu- 

10 tion of the test case. 

The invention provides a computer software product, 
including a computer-readable medium in which computer 
program instructions are stored, which instructions, when 
read by a computer, cause the computer to verify a design 

15 by generating a test case for execution using the design. 
One of the resources in the design is required at a pre- 
determined time to accommodate a signal during execution 
of the test case. The method is further carried out by 
designating the one resource as an unidentified resource, 

20 delaying binding the signal with the unidentified re- 
source until immediately prior to the predetermined time, 
thereafter binding the signal with the unidentified re- 
source to define a bound resource, and accessing the 
bound resource. 

25 The invention provides a verification system of veri- 

fying a design, including a test generator adapted to 
generate a test case for execution using the design, 
wherein the design includes a plurality of resources. One 
of the resources is required at a predetermined time to 

30 accommodate a signal during execution of the test case. 
The test generator is further adapted to designate the 
one resource as an unidentified resource, and to delay 
binding the signal with the unidentified resource until 
immediately prior to the predetermined time, thereafter 
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binding the signal and the unidentified resource to de- 
fine a bound resource, and accessing the bound resource. 

The invention provides a method of verifying a de- 
sign, which is carried out by generating a stream of in- 
5 structions, the stream including a first instruction that 
references an unidentified first resource, setting a 
first flag that designates the first resource as uniden- 
tified, copying the first resource to a second resource, 
setting a second flag that designates the second resource 

10 as unidentified, including the second resource in a set 
of unidentified resources that is associated with the 
first resource, wherein each member of the set has a 
status flag designating the member as unidentified, and 
thereafter identifying the second resource, clearing the 

15 second flag, and removing the second resource from the 
set. 

In one aspect of the method, identifying the second 
resource also includes identifying each member of the set 
with the second resource, and clearing the status flag of 
20 each member. 

Another aspect of the method includes accessing the 
second resource. 

In yet another aspect of the method, identifying the 
second resource includes requesting a content of the sec- 
25 ond resource. 

The invention provides a computer software product, 
including a computer-readable mediiom in which computer 
program instructions are stored, which instructions, when 
read by a computer, cause the computer to perform a 
30 method of verifying a design, which is carried out by 
generating a stream of instructions, the stream including 
a first instruction that references an unidentified first 
resource, setting a first flag that designates the first 
resource as unidentified, copying the first resource to a 
35 second resource, setting a second flag that designates 
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the second resource as unidentified, including the second 
resource in a set of unidentified resources that is asso- 
ciated with the first resource, wherein each meniber of 
the set has a status flag designating the member as uni- 
5 dentified, and thereafter identifying the second re- 
source, clearing the second flag, and removing the second 
resource from the set. 

The invention provides a verification system for 
verifying a design, including a test generator adapted to 

10 perform a method that includes generating a stream of in- 
structions, the stream including a first instruction that 
references an unidentified first resource, setting a 
first flag that designates the first resource as uniden- 
tified, copying the first resource to a second resource, 

15 setting a second flag that designates the second resource 
as unidentified, including the second resource in a set 
of unidentified resources that is associated with the 
first resource, wherein each member of the set has a 
status flag designating the member as unidentified, and 

20 thereafter identifying the second resource, clearing the 
second flag, and removing the second resource from the 
set . 

The invention provides a method of verifying a de- 
sign, which is carried out by generating a stream of in- 

25 struct ions for evaluation of a Boolean function in the 
design, constructing a set of inputs for the Boolean 
function, the set including members having unidentified 
input resources, and wherein the inputs are outcome de- 
terminative of the Boolean function. The method is fur- 

30 ther carried out by selecting one of the members, resolv- 
ing an identity of the selected member, excluding the se- 
lected member from the set of inputs, removing all re- 
maining members of the set of inputs that are no longer 
outcome determinative of the Boolean function, iterating 

35 the steps of selecting, resolving, excluding and removing 



9 



IL920030044US1 



until no more than one member remains in the set of in- 
puts, and determining the output resource as a copy of 
the one member. 

An aspect of the method includes setting a flag that 
5 designates the output resource as unidentified, and in- 
cluding the output resource in a set of unidentified re- 
sources that is associated with the one member, respec- 
tive status flags being associated with each member of 
the set of unidentified resources that designate an uni- 

10 dentified status thereof. 

An additional aspect of the method includes associat- 
ing a respective inversion flag with each of the input 
resources and the output resource that indicates an in- 
version status thereof, and setting the inversion flag of 

15 the output resource to a negation of the inversion flag 
of the input resource that corresponds to the one member. 

The invention provides a computer software product, 
including a computer-readable medium in which computer 
program instructions are stored, which instructions, when 

20 read by a computer, cause the computer to perform a 
method of verifying a design, which is carried out by 
generating a stream of instructions for evaluation of a 
Boolean function in the design, constructing a set of in- 
puts for the Boolean function, the set including members 

25 having unidentified input resources, and wherein the in- 
puts are outcome determinative of the Boolean function. 
The method is further carried out by selecting one of the 
members, resolving an identity of the selected member, 
excluding the selected member from the set of inputs, re- 

30 moving all remaining members of the set of inputs that 
are no longer outcome determinative of the Boolean func- 
tion, iterating the steps of selecting, resolving, ex- 
cluding and removing until no more than one member re- 
mains in the set of inputs, and determining the output 

35 resource as a copy of the one member. 
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The invention provides a verification system of veri- 
fying a design, including a test generator adapted to 
perform a method that includes generating a stream of in- 
structions for evaluation of a Boolean function in the 
5 design, constructing a set of inputs for the Boolean 
function, wherein input resources associated with members 
of the set of inputs are unidentified, and wherein the 
inputs are outcome determinative of the Boolean function. 
The method is further carried out by selecting one of the 

10 members, resolving an identity of the selected member, 
excluding the selected member from the set of inputs, re- 
moving all remaining members of the set of inputs that 
are no longer outcome determinative of the Boolean func- 
tion, and iterating the steps of selecting, resolving, 

15 excluding and removing until no more than one member re- 
mains in the set of inputs. 

The invention provides a verification system for a 
computer program-under- test , including a case generator 
that accepts the program-under-test as a program input, a 

20 simulator for executing the program-under- test responsive 
to values provided by the case generator, and a late 
binding component that accepts a request for user input 
responsively to the program-under-test, the late binding 
component being adapted to hold the request for user in- 

25 put in a memory, and for associating an unidentified re- 
source with the request for user input. The system fur- 
ther includes a tracking component for determining when 
the unidentified resource is actually required in the 
simulator, and a user interface linked to the tracking 

30 component for accepting information to satisfy the re- 
quest for user input, wherein responsively to the user 
interface the tracking component binds the information to 
the unidentified resource. 
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According to an aspect of the verification system, 
the case generator is further adapted to present a con- 
text of the program-under-test on the user interface. 

According to a further aspect of the verification 
5 system, the case generator is adapted to report an actual 
requirement for the user input in the simulator. 

The invention provides a method for verifying a com- 
puter program- under- test, which is carried out by accept- 
ing a request for user input responsively to the program- 

10 under-test, memorizing the request for user input, asso- 
ciating an unidentified resource with the request for 
user input, determining when the unidentified resource is 
actually required in an execution of the program-under- 
test, and thereafter accepting information from a user to 

15 satisfy the request for user input, binding the informa- 
tion to the unidentified resource to define a late-bound 
resource, and using the late-bound resource in the execu- 
tion of the program-under-test . 

The invention provides a computer software product, 

20 including a computer-readable medium in which computer 
program instructions are stored, which instructions, when 
read by a computer, cause the computer to perform a 
method for verifying a computer program-under-test, which 
is carried out by accepting a request for user input re- 

25 sponsively to the program-under- test, memorizing the re- 
quest for user input, associating an unidentified re- 
source with the request for user input, determining when 
the unidentified resource is actually required in an exe- 
cution of the program-under- test, and thereafter accept- 

30 ing information from a user to satisfy the request for 
user input, binding the information to the unidentified 
resource to define a late-bound resource, and using the 
late-bound resource in the execution of the program- 
under -test. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a better understanding of the present invention, 
reference is made to the detailed description of the in- 
vention, by way of example, which is to be read in con- 
5 jiinction with the following drawings, wherein like ele- 
ments are given like reference numerals, and wherein: 

Fig. 1 is a block diagram of a verification system 
that is constructed and operable in accordance with a 
disclosed embodiment of the invention; 
10 Fig. 2 is a diagram that illustrates a method of gen- 

erating an instruction having a write-read dependency in 
accordance with a disclosed embodiment of the invention; 

Fig. 3 is a flow diagram illustrating a method for 
implementation of late binding in accordance with a dis- 
15 closed embodiment of the invention; 

Fig. 4 shows a truth table of a simple AND gate with 
two inputs, A and B, and an output Y illustrating the 
principles of a disclosed embodiment of the invention; 

Fig. 5 is a high level block diagram of an I/O de- 
20 vice suitable for design verification in accordance with 
a disclosed embodiment of the invention; 

Fig. 7 is a table that compares the simulation time 
of two versions of the I/O device shown in Fig. 5, in ac- 
cordance with a disclosed embodiment of the invention; 
25 and 

Fig. 8 is a block diagram of a software verification 
system that is constructed and operable in accordance 
with a disclosed embodiment of the invention. 

DETAILED DESCRIPTION OF THE INVENTION 

30 In the following description, numerous specific de- 

tails are set forth in order to provide a thorough under- 
standing of the present invention. It will be apparent to 
one skilled in the art, however, that the present inven- 
tion may be practiced without these specific details. In 
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other instances well-known circuits, control logic, and 
the details of computer program instructions for conven- 
tional algorithms and processes have not been shown in 
detail in order not to unnecessarily obscure the present 
5 invention. 

Software programming code, which embodies aspects of 
the present invention, is typically maintained in perma- 
nent storage, such as a computer readable medium. In a 
client /server environment, such software programming code 

10 may be stored on a client or a server. The software pro- 
gramming code may be embodied on any of a variety of 
known media for use with a data processing system. This 
includes, but is not limited to, magnetic and optical 
storage devices such as disk drives, magnetic tape, com- 

15 pact discs (CD's), digital video discs (DVD's), and com- 
puter instruction signals embodied in a transmission me- 
diiam with or without a carrier wave upon which the sig- 
nals are modulated. For example, the transmission medium 
may include a communications network, such as the Inter- 

20 net. In addition, while the invention may be embodied in 
computer software, the functions necessary to implement 
the invention may alternatively be embodied in part or in 
whole using hardware components such as application- 
specific integrated circuits or other hardware, or some 

25 combination of hardware components and software. 

Definitions. 

The term ''variable" is used herein to mean the iden- 
tification, designation or evaluation of resources. The 
term *unidentif ied resource" refers to any element of a 

30 design-under- test that does not have a unique, predeter- 
mined value. In the context of a software design, the 
term ''variable" may also refer to an object. Typically, 
such resources include (but are not limited to) signals, 
registers, groups of signals and groups of registers in 

35 the design-under- test. The ''value" of a resource means a 
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specific identification of the resource for purposes of a 
particular operation carried out by the design, for exam- 
ple a field in an instruction. Alternatively, the term 
value may refer to the content of a field. ''Designation" 
5 has the same meaning here as identification. 

System Overview. 

Turning now to the drawings, reference is made to 
Fig. 1, which is a block diagram of a typical verifica- 
tion system 10 that is operable in accordance with a dis- 

10 closed embodiment of the invention. The teachings of the 
present invention are not restricted to systems that are 
configured like the verification system 10, but are ap- 
plicable to many testing systems that have different ar- 
chitectures. Typically, a design-under-test is verified 

15 through a suite of test cases to be generated. The solu- 
tion given here is also appropriate for many other test 
generation approaches, including manual test generation. 

The verification system 10, which can be used for 
verifying a software or hardware implementation, has sev- 

20 eral basic interacting components. 

The verification system 10 enables the creation of 
tests that have various degrees of randomness. The abil- 
ity of the verification system 10 to introduce random un- 
specified values is generally desirable, since design 

25 flaws in practice are usually unpredictable. 

A generic test case generator engine 16 has a user 
input 18, which influences the algorithms used to gener- 
ate test cases. 

Test cases 22 are executed by an execution engine 24 

30 on an implementation of the system-under-test. The execu- 
tion engine 24 can be a simulator of the system-under- 
test, or the system-under-test itself. The system-under- 
test can be a complex software implemented system, for 
example middleware, or a hardware simulator. Indeed, the 

35 system-under-test itself may be a simulator. 
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Execution of the test cases 22 produces a response 26 
from the system. The response 26 is typically submitted 
to a validation process, represented in Fig. 1 as a vali- 
dation engine 28, which has knowledge of the expected re- 
5 sponse, validates the response 26, cind produces valida- 
tion results 30. 

Late Binding* 

With continued reference to Fig. 1, the case genera- 
tor engine 16 is adapted to late binding of values. As 

10 noted above, generation of interesting test cases for 
hardware designs that reach dark-corners is a difficult 
task, even using dynamic generation, because often the 
state of the design changes before the generated stimuli 
affect the interesting area in the design. When this oc- 

15 curs, the reason that led to the generation of the spe- 
cific pattern no longer exists. A solution to this prob- 
lem according to the instant invention is to delay iden- 
tification of resources by the case generator engine 16 
to parts of the generated stimuli until these resources 

20 are actually used in the design. In other words, as long 
as an attribute having an undefined value is merely cop- 
ied or moved from one place to another, there is no need 
to set its value. It is not until an attribute with an 
undefined value is to be used for calculations or affects 

25 the control flow, that the case generator engine 16 is 
called to set the value of the attribute, shortly before 
the access is actually performed. This delayed assign- 
ment, or late binding, of values or resources allows the 
case generator engine 16 to have a more accurate view of 

30 the state of the design, and thus it can choose the re- 
source, or resolve the value, that more precisely leads 
to a desired event. 

Reference is now made to Fig. 2, which is a diagram 
that illustrates a method of generating an instruction 

35 having a write-read dependency in accordance with a dis- 
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closed embodiment of the invention. It is assumed, for 
the purpose of Fig. 2, that the design-under- test is a 
hardware processor having execution pipelines, and fetch 
and decode units. However, it is to be understood that 
5 the principles of the invention are not limited to such 
processors, but can be applied to other hardware designs. 
Indeed, the invention is applicable to the verification 
of software designs, as is explained in further detail 
hereinbelow. 

10 In a first stage 32, the case generator engine 16 

(Fig. 1) generates a new add instruction 34 in which the 
target register is a register R7 and the source registers 
are a register R4, and a second source register, labeled 
Rx, which is an as yet unidentified field of the instruc- 

15 tion. It should be noted that choosing the second source 
register based on the current state of the execution 
pipelines does not lead to the desired results, as will 
be shown hereinbelow. 

After the instruction 34 is fetched in a second 

20 stage 36, it flows for several cycles through the fetch 
and decode units, represented by a block 38. The source 
register field in the instruction is only copied from 
place to place in these units. Thus, the undefined value 
is copied with the rest of the instruction and its iden- 

25 tification is not established by the generator. 

In a third stage 40, the instruction 34 reaches the 
data fetch stage in an execution pipeline 42. In this 
stage, the register file is accessed with the source reg- 
ister to fetch the operand of the instruction. Since this 

30 access is not a simple copy, it triggers the test genera- 
tor to identify the source register. The case generator 
engine 16 (Fig. 1) examines the state of the pipelines 
and finds that an instruction MUL R17, R2, RO, at the 
execution stage in another execution pipeline 44 is writ- 

35 ing to a register R17. Therefore, the case generator en- 
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gine 16 identifies the source register Rx in the instruc- 
tion 34 as the register R17, and thereby creates the re- 
quired write-read dependency involving the register R17 . 
In a fourth stage 46, The final state of the pipe- 
5 lines 42, 44 is shown after the identity of the source 
register Rx of the instruction 34 has been established. 
The newly modified instruction 34 has been relabeled as 
instruction 48 in the stage 46. 

Referring again to the stage 32, when the instruc- 

10 tion 34 was created, potential target registers R7 and R6 
were engaged by instructions flowing through the pipe- 
lines 42, 44, respectively • None of these potential tar- 
get registers can be used to create the requested depend- 
ency. When simulating late binding in this manner, it is 

15 important to maintain precise simulation semantics in or- 
der to properly represent these registers. 

First Inplementation of Late Binding. 

Reference is now made to Fig. 3, which is a flow dia- 
gram illustrating a method for implementation of late 

20 binding, while maintaining correct simulation semantics 
in accordance with a disclosed embodiment of the inven- 
tion. The method, disclosed with reference to Fig. 3 for 
purposes of presentation, can be used in many verifica- 
tion systems, including the verification system 10 

25 (Fig. 1), in order to implement the technique of late 
binding that has been described with reference to Fig. 2. 

Each variable in the design is associated with a Boo- 
lean flag that indicates if its identification is unde- 
fined and a pointer to a set or linked list that contains 

30 all the other variables that should have the same identi- 
fication, respectively. Linked lists referenced by point- 
ers are a convenient way to maintain sets of variables. 
However, other techniques of implementing sets will occur 
to those skilled in the art, and can be used to carry out 

35 the invention. Access to a variable having its undefined 
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flag set during simulation using a test generator is han- 
dled in the following way. At initial step 50, a vari- 
able B is identified during simulation for further treat- 
ment, 

5 Control now proceeds to decision step 52, where it is 

determined if the variable B has its undefined flag set, 
meaning that its identity among, for example a set of 
registers, is currently not established. If the determi- 
nation at decision step 52 is negative, then the identity 

10 of the variable B is already known. Hence, the inventive 
late binding technique is not currently applicable to the 
variable B. Control proceeds to final step 54, and the 
procedure terminates. 

If the determination at decision step 52 is affirma- 

15 tive, then control proceeds to decision step 56, where it 
is determined if the variable B is being copied to an- 
other variable A. If the determination at decision 
step 56 is negative, then control proceeds to final 
step 54. 

20 If the determination at decision step 56 is affirma- 

tive, then control proceeds to step 58, where the unde- 
fined flag associated with the variable A is set. 

Next, at step 60, the variable A is added to a set of 
undefined variables that are currently associated with 

25 the variable B, all of which have their undefined flags 
set, and the set of undefined variables associated with 
the variable A is equated to the set of the variable B. 

Control now proceeds to decision step 62, where it is 
determined if a value is being assigned to the variable A 

30 in the simulation. If the determination at decision 
step 62 is negative, then control proceeds to decision 
step 64, which is described below. 

If the determination at decision step 62 is affirma- 
tive, then control proceeds to step 66, where the unde- 

35 fined flag of the variable A is cleared. 
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Next, at step 68, the variable A is removed from the 
sets of variables with undefined values with which it is 
currently associated. Control then proceeds to final 
step 54. 

5 Decision step 64 is performed when the determination 

at decision step 62 was negative. Here it is determined 
if a value is being assigned to the variable A by an ex- 
ternal tool, for example the test generator. When an ex- 
ternal tool assigns a particular value to a variable 

10 presently having an undefined value, it is desirable that 
the particular value also be associated with all the cop- 
ies, that is to all members of the set of undefined val- 
ues with which the variable is associated. The effect of 
such an assignment is equivalent to assigning this value 

15 primarily, instead of leaving the value undefined and em- 
ploying late binding. 

In contrast, at decision step 62, one is dealing with 
an assignment that is made by the design itself. This is 
a local assignment that overrides the undefined value. 

20 Therefore, in this case the assignment is limited to one 
variable, while all copies still remain undefined. 

If the determination at decision step 64 is negative, 
then control proceeds to decision step 70, which is de- 
scribed below. 

25 If the determination at decision step 64 is affirma- 

tive, then control proceeds to step 72. Here the identity 
that was assigned to the variable A in decision step 64 
is assigned to each member of the set of undefined vari- 
ables that is associated with the variable B. 

30 Next, at step 74, the undefined flag of each member 

of the set of undefined variables that is associated with 
the variable B is cleared. Control then proceeds to final 
step 54. 

Decision step 70 is performed when the result of the 
35 determination at decision step 64 is negative. A determi- 
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nation is made whether the variable A is being accessed 
for a purpose other than copying, for example in the 
course of an assignment, for example C := A + D. 

If the determination at decision step 70 is negative, 
5 then control proceeds to final step 54. 

If the determination at decision step 70 is affirma- 
tive, then control proceeds to step 76. Here late binding 
of the variable A occurs. A value is assigned to the 
variable A. That is to say, the variable A is identified 
10 with a member of its group, such as a particular regis- 
ter, or the content of the variable A is resolved. 

Next, step 78 and step 80 are performed on the vari- 
able A in the same manner as step 72 and step 74, respec- 
tively. The details are not repeated in the interest of 
15 brevity. 

Next, at step 82 the variable A is accessed, for ex- 
ample in order to accomplish the purpose noted in deci- 
sion step 70. Control then proceeds to final step 54. 

Second Implementation o£ Late Binding. 

20 In the method disclosed with reference to Fig. 3, 

late binding is supported, while maintaining correct 
simulation semantics. An undefined value or identifica- 
tion can exist as long as the value is merely copied or 
moved from place to place. If a resource with an unde- 

25 fined identity is involved in any other operation, the 
case generator engine 16 (Fig. 1) is called and the value 
of the variable is resolved. This method is useful in 
high-level descriptions, such as transaction level mod- 
els, where many assignments exist and the operations on 

30 resources may be complex. As the level of description 
gets lower, the number of pure copy operations gets 
smaller. Therefore, the possibility that an undefined re- 
source will survive and reach areas deep in the design is 
reduced. As a result, the chances that the example of 

35 Fig. 2 will work in a register transfer level (RTL) de- 
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scription of a processor are practically zero. On the 
other hand, many operations in RTL descriptions are sim- 
ple Boolean functions such as AND, OR, NOT, and XOR. 
Therefore, the method disclosed in Fig. 3 could not be 
5 efficiently employed as to these Boolean functions. 

Reference is now made to Fig. 4, which shows a truth 
table of a simple AND gate with two inputs, A and B, and 
an output Y. The two left columns in the table show the 
inputs to the gate before the output is evaluated. The 

10 other columns show the inputs and output of the gate af- 
ter the output is evaluated. A notation Ux represents an 
undefined value, and a notation Rx represents an unde- 
fined value that was resolved, that is, the input x was 
assigned by the case generator engine 16 (Fig. 1) . The 

15 middle section of the table shows the evaluation of the 
gate according to the basic method disclosed above with 
respect to Fig. 3. Since operation of the AND gate is not 
a simple copy operation, each time the output is evalu- 
ated, the values of the inputs are resolved before the 

20 actual evaluation occurs, according to the basic method. 
For example, the third non-header row in the table shows 
a case in which the value of the input A is 0, while the 
value of the input B is undefined. In this case, accord- 
ing to the basic method, the value of the input B is re- 

25 solved before the output is evaluated, although the value 
of the input B does not affect the value of the output Y. 

The need to resolve the value of the input B in the 
example above can be avoided by using lazy resolution; 
that is, by delaying resolution of undefined values until 

30 they are actually needed to determine the value of the 
evaluated output. In the example above, the value 0 of 
the input A determines the value of the output Y, and 
therefore, using lazy resolution, there is no need to re- 
solve the undefined value of the input B. 
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When one of the inputs to the AND gate has the 
value 1, the gate just copies the value of the other in- 
put to its output. Following the spirit of the basic al- 
gorithm, if the value of the other input is undefined, it 
5 should be copied to the output of the gate. This ''smart" 
resolution scheme is shown in the right section of the 
table in Fig. 4. Using smart resolution, undefined values 
at the inputs of an AND gate need to be resolved only if 
both inputs are undefined and neither input is a copy of 

10 the other. For example, if at an earlier time an assign- 
ment A := B was executed, then the variable A is a copy 
of the variable B, and no evaluation is needed when using 
smart resolution. Whether the variables are copies can be 
determined by consulting the linked list of one of them. 

15 If the other variable is found in the list, then the two 
variables are copies. This scheme can be further improved 
by resolving just one of the inputs to the gate when both 
inputs are undefined. If this resolved input is determi- 
native of the output, then there is no need to resolve 

20 the other input. 

Modifications of the smart resolution technique dis- 
closed above for use with other logical gates will be ap- 
parent to those skilled in the art. For example, in an OR 
gate, if one of the inputs has a value 1, which is out- 

25 come determinative, there is no need to resolve the value 
of the other input, should it be presently undefined. 

To handle inverters and other inverting gates, a flag 
is associated with each attribute. This flag indicates 
whether or not the undefined value in the attribute is 

30 inverted, compared to the head of the set of attributes 
having the same undefined value. When an inverter with 
undefined input is evaluated, the output is added to the 
undefined set of the input as explained above with re- 
spect to step 60 (Fig. 3), and the inverted flag of the 
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output is set to the negation of the inverted flag of the 
input . 

Using the ''smart" resolution scheme and the inverter 
handling, it is possible to handle all Boolean functions 
5 with two inputs without any resolution, unless both in- 
puts are undefined. It is also possible to handle more 
complex entities that are often used in RTL descriptions, 
such as multiplexers. 

Listing 1 is a pseudocode fragment, which describes a 

10 generic method for use with a test generator that handles 
arbitrary Boolean functions. The method starts by con- 
structing a set C of the inputs with an undefined value, 
to which the output is sensitive, i.e., the value of the 
output may be affected by the values of these inputs. 

15 Next, the an input c is chosen in the set C and the case 
generator is called to resolve its value. There are many 
possible heuristics for choosing the input c. For exam- 
ple, choosing the input that creates the greatest reduc- 
tion in the set C, choosing the input that is less needed 

20 to reach a required event, etc. After the input c is re- 
solved, it is removed from the set C. In addition, all 
other inputs in the set C, to which the output is no 
longer sensitive, are also removed from the set C. These 
steps are continued until at most one input remains in 

25 the set C. If the set C is empty, the value of the output 
is determined. If one input* is left in the set C, then 
the output is either a copy or a negation of the remain- 
ing input. In this case, the undefined flag of the output 
is set and the output is added to the undefined value set 

30 of the remaining input. 

Exantple 1. 

The method disclosed with reference to Fig. 3 has 
been tested using the Odette Verification Environment, 
details of which are available from the European Elec- 
35 tronic Chips & Systems design Initiative, 2 Avenue de 
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Vignate, Pare Equation, 38610 Gieres, France. The Odette 
Verification Environment provides verification support 
for object-oriented designs. It is built on top of the 
SystemC class library, available from Open SystemC Ini- 
5 tiative, 1177 Branham Lane #302, San Jose, CA, 95118- 
3766, The current prototype has been implemented outside 
the simulation kernel of SystemC. Therefore, it supports 
late binding only for a small set of types, namely ob- 
jects and data-members of objects that are built-in C++ 
10 types. 

Reference is now made to Fig. 5, which is a high 
level block diagram of an I/O device 84, the design of 
which was verified using the current prototype in order 
to evaluate the effectiveness of the late binding tech- 

15 nique and its ability to improve test coverage. The de- 
vice 84 receives requests through four channels 86, 88, 
90, 92 and processes the requests using two processing 
elements 94, 96. The processing element that processes 
each request is determined by the type of the request. 

20 Each channel, upon receiving a request, checks its valid- 
ity and sends it to the proper processing element. Re- 
quests are stored in a queue until the processing element 
can handle them. The processing elements 94, 96 read the 
requests in their respective queues one at a time, and 

25 process them. Upon completion of the service, a response 
is sent to the requesting channel. 

Because the processing elements 94, 96 seirvice re- 
quests independently, and possibly at different rates, it 
is possible that requests are serviced out-of-order 

30 (i.e., two requests received in the channel 86 could be 
handled by different processing elements 94, 96, and 
could be serviced in the reverse order from which they 
were received) . The goal of the verification in this ex- 
ample was to cover all possible out-of-order events that 

35 could occur in the device 84. 
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Three types of generation schemes were compared: 
static generation, dynamic generation, and dynamic gen- 
eration with late binding. The generation strategy in all 
the schemes was to send a first request to the processing 
5 element having the longer queue, followed by a second re- 
quest to the other processing element. Since there is a 
high probability that the first request would experience 
a longer delay while waits in the longer queue than the 
second request, the likelihood of an out-of-order event 
10 would be significant. 

The implementation of this strategy in the three gen- 
eration schemes was as follows: 

Using static generation, the generator generated a 
sequence of recjuests to one of the processing elements in 
15 an attempt to fill its queue. It then generated a request 
to the other processing element. 

Using dynamic generation, the generator examined the 
state of the two queues before a request was generated, 
and set the type of the request to a category required to 
20 be handled by the processing element having the longer 
queue. The next request was sent to the other processing 
element . 

When late binding was used, the value of the request 
type, which, as noted above, determines the processing 

25 element in which it is to be served, was set to be unde- 
fined. When the input channel finished its local process- 
ing of the request, it attempted to select the processing 
element to service the request according to the request 
type. Since the value of the type was undefined at this 

30 stage, the generator was called to resolve it. At this 
stage, the generator had an accurate view of the state of 
the queues in the two processing elements 94, 96, and it 
set the request type to a value requiring service of the 
request by the processing unit having the longer queue. 

35 The input channel accordingly selected this processing 



26 



IL920030044US1 



unit. The next request was sent to the other processing 
element . 

Reference is now made to Fig. 6, which is a graph 
comparing coverage of out-of-order events for the three 
5 generation schemes. The graph shows that 100% of possible 
out-of-order events were covered by dynamic generation 
with late binding in less than 300 test-cases, while dy- 
namic and static generation were able to reach only 75% 
and 60% of the out-of-order events, respectively. 
10 Simulation times of test cases on two models of the 

device 84 that differed in their level of abstraction are 
illustrated. In one model the handling of the transac- 
tions both in the channels and the processing elements 
was done using simple functions, while in the second mod- 
15 els the implementation of these functions was more de- 
tailed and was closer to an actual low-level implementa- 
tion. Measurements were taken for three cases: 

1. No late binding. That is, the late binding op- 
tion was turned off. 
20 2, Late binding was turned on, but was not used. 

That is, some attributes in the requests could 
have undefined values, but the generator never 
set the values of the attributes to be undefined. 
3. Late binding was used. The generator followed 
25 the generation strategy for late binding de- 

scribed above. 

Reference is now made to Fig. 7, which is a table 
that compares the simulation time of these three cases 
for both models of the I/O device in accordance with a 

30 disclosed embodiment of the invention. The table shows 
that simply activating late binding slowed the simulation 
by 10%. Using the late binding capabilities increased the 
slowdown to 30%. Most of delay can be attributed to time 
spent in the generator itself, and only a small part is 

35 spent in maintenance of the sets of attributes having the 
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same undefined values. Therefore, it is expected that 
slowdown will become less dominant as the simulation rate 
decreases and simulation performance becomes more impor- 
tant. For example, while simulation time more than dou- 
5 bled in the second model (for the same test-cases), the 
overhead of using late binding increased from 5.2 seconds 
to 6.5 seconds, or by just 25%. 

Alternative Embodiment. 

The techniques of late binding can be used advanta- 

10 geously to improve efficiency of both concurrent and se- 
quential software applications. Conventionally, a soft- 
ware application can prompt the user for information to 
be assigned to a resource, which is typically a variable, 
the identity of which is resolved at the time of the 

15 user's input. However, in some circumstances the program 
never needs to use the information, and thus never needs 
to access this resolved variable. However in all cases, 
using the conventional approach, the variable is unavail- 
able in the pool, and hence the application is to that 

20 extent, deprived of a resource. Referring again to 
Fig. 3, in the context of a software design, step 76 is 
modified to include prompting the user for information, 
and associating this information with the variable A at 
the time the variable A is resolved. Similarly, using the 

25 techniques of lazy resolution disclosed above with refer- 
ence to Listing 1 , the user is prompted for information 
only when it is necessary to resolve the input c. The ad- 
vantages of this technique are twofold. First, the user 
is spared the burden of inputting information unnecessar- 

30 ily, and secondly the design operates with greater effi- 
ciency. 

Reference is now made to Fig. 8, which is a block 
diagram of a software verification system 98, that is 
adapted to software testing and is constructed and oper- 
35 able in accordance with a disclosed embodiment of the in- 
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vention. The operation of the software verification sys- 
tem 98 is similar to that of the verification system 10 
(Fig. 1), except now the input is a program-under- 
test 102. A case generator 104 is responsible for provid- 
5 ing variables to a simulator 106, which actually executes 
the program-under-test 102 . 

The case generator 104 is provided with a late bind- 
ing component 108. During execution, the program-under- 
test 102 may request a particular input to be assigned to 

10 a variable. However, this request is not immediately pre- 
sented to the users, but instead is held in abeyance in a 
buffer within a tracking component 110. Subsequently, 
when an unidentified variable that would contain the par- 
ticular input is actually required in the simulation, the 

15 tracking component 110 causes the case generator 104 to 
generate a request for a user to provide the particular 
input via a user interface 112. The user interface 112 is 
configured in some embodiments to present the context in 
which the requested input is to be used. Responsively to 

20 the user input, the late binding component 108 binds the 
input to the variable. The bound variable is then commu- 
nicated to the simulator 106. 

The case generator 104 is aware of program context, 
and in many cases is capable of generating the requested 

25 input with a high probability of being correct. In some 
embodiments, the case generator 104 reports via the user 
interface 112 whether the requested input was ever actu- 
ally needed, or merely remained in abeyance during the 
simulation. 

30 Exasiiple 2. 

The following pseudocode illustrates one mode of op- 
eration of the software verification system 98. 

input X 

35 
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Much later first use of x 
if (X == 87654 ) then { 
Show it to a user OR 
5 Show it to a generator 

} 



Here the generator is context-aware. It knows that if 
X is used in a condition, generate values such that in 
10 one case x == 87654 is true and in another case the con- 
dition is false. So with high probability the generator 
generates x as 87654, which otherwise would have negligi- 
ble probability. This obviously improves the testing. 



15 Second execution, 

input X 



Much later first use of x, 
20 which is different from the first execution, 

if (x == y ) then 



25 In the second execution, use of the variable x does 

not occur in same location in the program as in the first 
execution, as it depends on other inputs. Assume that y 
is now 99. The generator will put x = 99 with a high 
probability of correctness, (as the user would) . 

30 The binding of the variable x is dynamic, and is bet- 

ter then a static analysis, which could have deter- 
mined 87654 to be a good value from a general understand- 
ing of the program. 

Methods and systems have been presented herein to in- 

35 crease the ability of test generators to reach obscure 
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cases in the design by delaying the assignment of values 
in the generated stimuli until they are needed by the de- 
sign. This late binding technique allows a test generator 
to have a more accurate view of the state of the design, 
5 and thus it can more precisely choose the values that 
lead to a desired event. Experimental results indicate 
that late binding can significantly improve coverage with 
a reasonable penalty in simulation time. It will be ap- 
preciated by persons skilled in the art that the present 

10 invention is not limited to what has been particularly 
shown and described hereinabove. Rather, the scope of the 
present invention includes both combinations and sub- 
combinations of the various features described herein- 
above, as well as variations and modifications thereof 

15 that are not in the prior art, which would occur to per- 
sons skilled in the art upon reading the foregoing 
description. 



COMPUTER PROGRAM LISTINGS 

20 Listing 1 



while IC l> 1 do { 

Choose an input c from C; 
Resolve the value of c; 
25 Remove from C any inputs to which the output is no 
longer sensitive; 
if IC 1= 1 then 

The value of the output is undefined; 
The output is added to the undefined value set of 
30 the input in C; 
else 

The value of the output is determined; 

}. 



IL920030044US1 



